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Description 

Technical field of the Invention 

5 [0001] The present invention relates to the field of image processing. More particularly, the present invention relates 
to display assennblies, computer program products and methods for defect compensation and/or masking. 

Background of the invention 

10 [0002] Matrix based or matrix addressed displays are composed of individual image forming elements, called picture 
elements (or "pixels"), that can be driven (or addressed) individually by proper driving electronics. The driving signals 
can switch a pixel to a first state, the on-state (at which luminance is emitted, transmitted or reflected), to a second state, 
the off-state (at which no luminance is emitted, transmitted or reflected or for some displays, one or any intermediate 
state between on or off (modulation of the amount of luminance emitted, transmitted or reflected), as for example 

15 described in EP 0462619. 

[0003] Since matrix addressed displays are typically composed of many millions of pixels, very often pixels exist that 
are stuckin a certain state (on, off or anything in between). Where pixel elements comprise multiple sub-pixels, individually 
controllable or not, then one or more of the sub-pixel elements may become stuck in a certain state. For example, a 
pixel structure may comprise three sub-pixel elements for red, green and blue colors respectively. If one of these sub- 

20 pixel elements becomes stuck in a certain state, then the pixel structure has a permanent color shift. Mostly such problems 
are due to a malfunction in the driving electronics of the individual pixel (for instance, a defective transistor). Other 
possible causes are problems with various production processes involved in the manufacturing of the displays, and/or 
by the physical construction of these displays, each of them being different depending on the type of technology of the 
electronic display under consideration. 

25 [0004] It is also possible that a pixel or sub-pixel element is not really stuck in a state, but shows a luminance or color 
behavior that is significantly different from the pixels or sub-pixels in its neighbourhood. In some illustrative but non- 
limiting examples, a defective pixel shows a luminance behaviorthat differs more than 20% (at one or more video levels) 
from the pixels in its neighborhood, or a defective pixel shows a dynamic range (maximum luminance / minimum lumi- 
nance) that differs more than 15% from the dynamic range of pixels in its neighborhood, or a defective pixel shows a 

30 color shift greater than a certain value comparing to an average or desired value for the display. Of course other rules 
are possible to determine whether a pixel or sub-pixel is defective or not, and any condition that has a potential danger 
for image misinterpretation can be expressed in a rule to determine whether a pixel is a defective pixel. Bright or dark 
spots due to dust for example may also be considered as pixel defects. 

[0005] Defective pixels or sub-pixels are typically very visible for the user of the display. They result in a significantly 
35 lower (subjective) image quality and can be very annoying or disturbing forthe display-user. For demanding applications 
(such as medical imaging, in particular mammography) the defective pixels or sub-pixels can even make the display 
unusable forthe intended application, as it can result in an incorrect interpretation of the image being displayed. For 
applications where image fidelity is required to be high, such as for example in medical applications, this situation is 
unacceptable. 

40 [0006] U.S. Pat. No. 5,504,504 describes a method and display system for reducing the visual impact of defects 
present in an image display. The display includes an array of pixels, each non-defective pixel being selectively operable 
in response to input data by addressing facilities between an "on" state, whereat light is directed onto a viewing surface, 
and an "off' state, whereat light is not directed onto the viewing surface. Each defective pixel is immediately surrounded 
by a first ring of compensation pixels adjacent to the central defective pixel. The compensation pixels are immediately 

45 surrounded by a second ring of reference pixels spaced from the central defective pixel. The addressing circuit-determined 
value of at least one compensation pixel in the first ring surrounding the defective pixel is changed from its desired or 
intended value to a corrective value, in order to reduce the visual impact of the defect. In one embodiment, the value of 
the compensation pixels is selected such that the average visually defected value for all of the compensation pixels and 
the defective pixel is equal to the intended value of the defective pixel. In another embodiment, the values of the 

50 compensation pixels are adjusted by adding an offset to the desired value of each compensation pixel. The offset is 
chosen such that the sum of the offset values is equal to the intended value of the defective pixel. One potential disad- 
vantage of the solution proposed in U.S. Pat. No. 5,504,504 is that a trial and error method is required for every other 
display in order to obtain a reasonable correction result. 

55 Summary of the Invention 

[0007] It is an object of embodiments of the present invention to provide good apparatus or methods for displaying 
images on a display assembly. It is an, advantage of embodiments according to the present invention that good systems 
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and methods are provided for reducing tine visual innpact of defects present in an innaging layer of a display assembly. 
[0008] The above objective is accomplished by a method and device according to the present invention. 
[0009] In a first aspect, a display assembly is provided comprising a first imaging layer and a second imaging layer, 
wherein a drive signal to a pixel on one of the first and second imaging layers is based on a defect in the other of the 
5 first and second imaging layers. At least one ofthe first and second imaging layers may be a spatially modulated backlight. 
[0010] The assembly may include a third imaging layer disposed between said first and second imaging layers. Each 
of said first and third imaging layers may comprise a liquid crystal panel. 

[0011] The display assembly may comprise a first panel in the first imaging layer and a second panel in the second 

imaging layer. 

10 [0012] At least one of the first and second panels may be a liquid crystal display panel. 
[0013] The resolution of the second panel may differ from the resolution of the first panel. 

[0014] The display assembly may include a backlight and the second panel may be disposed between the first panel 
and the backlight. 

[0015] The present invention also relates to a method for displaying images in a display assembly comprising at least 
15 a first and second imaging layer, the method comprising providing a drive signal to a pixel on one of the first and second 
imaging layers based on a defect in the other of the first and second imaging layers. 

[0016] The method may comprise modulating pixel data in a neighbourhood around a defect of a display panel in a 
stack of at least two panels, wherein the neighborhood includes at least one pixel of another panel in the stack. 
[0017] The present invention furthermore relates to a method of image display, said method comprising modulating 
20 pixel data in a neighborhood around a defect of a display panel in a stack of at least two panels, wherein the neighborhood 
includes at least one pixel of another panel in the stack. 

[0018] In the above methods, the neighbourhood may only include pixels that lie along a single straight line in space. 
[0019] In the above methods, the method may comprise calculating a pixel drive signal according to a model of a 
response of a human visual system. 
25 [0020] In the above methods, the line may be perpendicular to the display plane of the panel. 

[0021 ] The present invention also relates to a control unit for controlling the displaying of images in a display assembly 
according to the methods for displaying as described above. The different method steps thereby may be implemented 
in hardware or software components adapted for performing these method steps. 

[0022] The present invention also relates to an autostereoscopic display assembly comprising a display panel and a 
30 view splitter, wherein said view splitter is configurable during operation according to a defect of the display panel. 
[0023] The view splitter may include a parallax barrier. The parallax barrier may comprise a liquid crystal panel. 
[0024] The view splitter may include an array of controllable lenses. 

[0025] At least one of the array of controllable lenses may comprise a material whose index of refraction is electrically 
controllable. 

35 [0026] The display may include a head-tracking device, and the view splitter may be configurable during operation 
according to an output of said head-tracking device. 

[0027] A principal surface of the view splitter may be disposed parallel to a display plane of the display panel. 
[0028] The view splitter may be disposed in front of a viewing surface of the display panel. 

[0029] The present invention also relates to a computer program product for, when executed on a computing means, 
40 performing a method for displaying images in a display assembly as described above or a method for reducing the visual 
impact of defects as described herein. 

[0030] The present invention also relates to a machine readable data storage device storing such a computer program 
product and/or the transmission thereof over a local or wide area telecommunications network. 

[0031] In one aspect, a method for reducing the visual impact of defects present in a matrix display comprising a 
45 plurality of display elements includes acts of providing a representation of a human vision system; characterizing at least 
one defect present in the display, the defect being surrounded by a plurality of non-defective display elements; deriving 
drive signals for at least some of the plurality of non-defective display elements in accordance with the representation 
ofthe human vision system and the characterizing ofthe at least one defect, to thereby minimize an expected response 
of the human vision system to the defect; and driving at least some of the plurality of non-defective display elements 
50 with the derived drive signals. 

[0032] Minimizing the response of the human vision system to the defect may comprise changing the light output value 
of at least one non-defective display element surrounding the defect in the display. Characterizing at least one defect 
present in the display may comprise storing characterization data characterizing the location and non-linear light output 
response of individual display elements, the characterization data representing light outputs of an individual display 
55 element as a function of its drive signals. 

[0033] Such a method may also include generating the characterization data from images captured from individual 
display elements. Generating the characterization data may comprise building a display element profile map representing 
characterization data for each display element of the display. 
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[0034] Providing a representation of tine human vision systenn may comprise calculating an expected response of a 
human eye to a stimulus applied to a display element. For calculating the expected response of a human eye to a 
stimulus applied to a display element, use may be made of a point spread function of the eye. The point spread function 
may be described analytically, for example based on using any of Taylor, Seidel or Zernike polynomials, or the point 
spread function may be described numerically. 

[0035] Some such methods are configured to take boundary conditions into account when minimizing the response 
of the human vision system to the defect. Minimizing the response of the human vision system may be carried out in 
real-time or off-line. A defect may be caused by a defective display element or by an external cause, such as dust 
adhering on or between display elements for example. 

[0036] Another aspect describes a system for reducing the visual impact of defects present in a matrix display com- 
prising a plurality of display elements and intended to be looked at by a human vision system. First characterization data 
for a human vision system is provided, and the system includes a defect characterizing device for generating second 
characterization data for at least one defect present in the display, the defect being surrounded by a plurality of non- 
defective display elements; a compensation and/or masking device for deriving drive signals for at least some of the 
plurality of non-defective display elements in accordance with the first characterization data and the second characterizing 
data, to thereby minimize an expected response of the human vision system to the defect; and means for driving at least 
some of the plurality of non-defective display elements with the derived drive signals. 

[0037] The compensation and/or masking device may comprise means to change the light output value of at least 
one non-defective display element surrounding the defect in the display. The defect characterizing device may comprise 
an image capturing device for generating an image of the display elements of the display. The defect characterizing 
device may also comprise a display element location identifying device for identifying the actual location of individual 
display elements of the display. In such a system, for providing the first characterization data, a vision characterizing 
device having calculating means for calculating the response of a human eye to a stimulus applied to a display element 
may be provided. 

[0038] In another aspect, a matrix display device for displaying an image intended to be looked at by a human vision 

system includes a plurality of display elements; a first memory for storing first characterization data for a human vision 
system; a second memory for storing second characterization data for at least one defect present in the display device; 
and a modulation device for modulating, in accordance with the firstcharacterization data and the second characterization 
data, drive signals for non-defective display elements surrounding the defect so as to reduce the visual impact of the 
defect present in the matrix display device. In such a matrix display device, the first and the second memory may 
physically be a same memory device. 

[0039] Another aspect describes a control unit for use with a system for reducing the visual impact of defects present 
in a matrix display comprising a plurality of display elements and intended to be looked at by a human vision system. In 
this example, the control unit includes a first memory for storing first characterization data for a human vision system; 
a second memory for storing second characterization data for at least one defect present in the display; and modulating 
means for modulating, in accordance with the first characterization data and the second characterization data, drive 
signals for non-defective display elements surrounding the defect so as to reduce the visual impact of the defect. 
[0040] In many cases, such arrangements may be applied to solve the problem of defective pixels and/or sub-pixels 
in matrix displays by making them almost invisible forthe human eye under normal usage circumstances. Such an effect 
may be achieved by changing the drive signal of non-defective pixels and/or sub-pixels in the neighborhood of the 
defective pixel or sub-pixel. 

[0041] It is an advantage of embodiments according to the present invention to provide good systems and methods 
for visually masking of pixel or sub-pixel defects present in a wide range of matrix addressed electronic display devices, 
especially fixed format displays such as plasma displays, field emission displays, liquid crystal displays, electrolumines- 
cent (EL) displays, light emitting diode (LED) and organic light emitting diode (OLED) displays, especially flat panel 
displays used in projection or direct viewing concepts. 

[0042] It is an advantage of embodiments according to the present invention that methods and systems are provided 
that can be applied to both monochrome and color displays and to emissive, transmissive, reflective and trans-reflective 
display technologies fulfilling the feature that each pixel or subpixel is individually addressable. 
[0043] It is an advantage of embodiments according to the present invention that methods and devices are provided 
for making pixel defects less visible and thus avoiding wrong image interpretation. The range includes methods usable 
for different types of matrix displays without a trial and error method being required to obtain acceptable correction results. 
[0044] Particular and preferred aspects of the invention are set out in the accompanying independent and dependent 
claims. Features from the dependent claims may be combined with features of the independent claims and with features 
of other dependent claims as appropriate and not merely as explicitly set out in the claims. 

[0045] The above and other characteristics, features and advantages of the present invention will become apparent 

from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way 
of example, the principles of the invention. This description is given for the sake of example only, without limiting the 
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scope of the invention. Tlie reference figures quoted below refer to tine attached drawings. 
Brief description of tlie drawings 
5 [0046] 

FIG. 1a Illustrates a matrix display having grayscale pixels with equal lunninance, and FIG. 1 b illustrates a matrix 
display having grayscale pixels with unequal luminance. 

FIG. 2a illustrates an LCD display having an RGB-stripe pixel arrangement: one pixel comprises three colored sub- 
10 pixels in stripe ordering, and the display has a defective green sub-pixel that is always fully on, and a defective red 

sub-pixel that is always off. 

FIG. 2b illustrates a greyscale LCD based matrix display having unequal luminance in sub-pixels. 

FIG. 3a illustrates an analytical point spread function (PSF) in case the optics is considered to be diffraction -limited 

only. 

15 FIG. 3b and FIG. 3c illustrate numerical PSFs that are measured on test subjects. 

FIG. 4a shows the eye response to a single pixel defect in the image plane if no masking is applied. FIG. 4b shows 
the eye response to the same pixel defect but after masking with 24 masking pixels has been applied. FIG. 4c shows 
the centre locations of the PSFs in the image plane of the masking pixels and the pixel defect. 
FIG. 5a illustrates nine pixels each having three sub-pixels and two domains. FIG. 5b shows one of such pixels in 

20 detail. 

FIG. 6 illustrates the transfomnation from a driving level to a luminance level. 

FIG. 7a shows a real green sub-pixel defect present in a display, and FIG. 7b shows the same green sub-pixel 
defect and artificial red and blue subpixel defects introduced to retain a color co-ordinate of the pixel which is as 
close to the correct color co-ordinate as possible. 
25 FIG. 8 illustrates possible locations for a real-time system for defect compensation and/or masking according to any 

embodiment of the present invention. 

FIG. 9 shows eight different examples of subpixel geometries that may be used in a color display panel. 
FIG. 10 shows a cross-section of a portion of a two-panel stack. 
FIG. 1 la and FIG. 1 1 b show two different configurations of LED backlight sources. 
30 FIG. 12a shows a backlight source and one pixel of each of two aligned panels in a two-panel display. 

FIG. 12b shows a cross-section of an example of a two-panel display in which a backlight source is not perfectly 
collimated. 

FIG. 12c shows an example of backlight sources having overlapping radiation patterns. 
FIG. 13a and 13b show two example configurations of fluorescent backlight sources. 
35 FIG. 14a shows an example of a fixed parallax barrier disposed in front of a display panel. 

FIG. 14b shows an example of a dynamic parallax barrier disposed in front of a display panel. 
FIG. 15 shows an example of a fixed lens structure disposed in front of a display panel. 

FIG. 1 6a shows an example of diversion, by a lens or other optical structure, of light emanating from adjacent pixels 

of a display panel to form two separate views. 
40 FIG. 1 6b shows one example of a generation, by a lens or other optical structure, of four different views from adjacent 

pixels of a display panel. 

FIG. 17 shows a schematic representation of a computing system as may be used for performing a method for 
displaying images or a method for reducing the visual impact of defects present in a matrix display according to 
embodiments of the present invention. 

45 

[0047] In the different figures, the same reference signs refer to the same or analogous elements. 

Description of illustrative embodiments 

50 [0048] The present invention will be described with respect to particular embodiments and with reference to certain 
drawings but the invention is not limited thereto but only by the claims. The drawings described are only schematic and 
are non-limiting. In the drawings, the size of some of the elements may be exaggerated and not drawn on scale for 
illustrative purposes. The dimensions and the relative dimensions do not correspond to actual reductions to practice of 
the invention. 

55 [0049] Furthermore, the terms first, second, third and the like in the description and in the claims, are used for distin- 
guishing between similar elements and not necessarily for describing a sequence, either temporally, spatially, in ranking 
or in any other manner. It is to be understood that the terms so used are interchangeable under appropriate circumstances 
and that the embodiments of the invention described herein are capable of operation in other sequences than described 
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or illustrated herein. 

[0050] Moreover, the terms top, bottom, over, under and the like in the description and the claims are used for descriptive 
purposes and not necessarily for describing relative positions. It is to be understood that the terms so used are Inter- 
changeable under appropriate circumstances and that the embodiments of the invention described herein are capable 

5 of operation in other orientations than described or illustrated herein. 

[0051] It is to be noticed that the term "comprising", used in the claims, should not be interpreted as being restricted 
to the means listed thereafter; it does not exclude other elements or steps. It is thus to be interpreted as specifying the 
presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or 
addition of one or more other features, integers, steps orcomponents, or groups thereof. Thus, the scope of the expression 

10 "a device comprising means A and B" should not be limited to devices consisting only of components A and B. It means 
that with respect to the present invention, the only relevant components of the device are A and B. 
[0052] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular 
feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment 
of the present invention. Thus, appearances of the phrases "in one embodiment" or "in an embodiment" in various places 

15 throughoutthis specification are not necessarily all referringtothesame embodiment, but may. Furthermore, the particular 
features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary 
skill in the art from this disclosure, in one or more embodiments. 

[0053] Similarly it should be appreciated that in the description of exemplary embodiments of the invention, various 
features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the 

20 purpose of streamlining the disclosure and aiding in the understanding of one or more of the various Inventive aspects. 
This method of disclosure, however. Is not to be Interpreted as reflecting an intention that the claimed Invention requires 
more features than are expressly recited In each claim. Rather, as the following claims reflect, inventive aspects lie in 
less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description 
are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate 

25 embodiment of this invention. 

[0054] Furthermore, while some embodiments described herein include some but not other features included in other 
embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and 
form different embodiments, as would be understood by those In the art. For example, in the following claims, any of 
the claimed embodiments can be used in any combination. 

30 [0055] Furthermore, some of the embodiments are described herein as a method or combination of elements of a 
method that can be implemented by a processor of a computer system or by other means of carrying out the function. 
Thus, a processor with the necessary instructions for carrying out such a method or element of a method forms a means 
for carrying out the method or element of a method. Furthemnore, an element described herein of an apparatus embod- 
iment is an example of a means for carrying out the function performed by the element for the purpose of carrying out 

35 the invention. 

[0056] In the description provided herein, numerous specific details are set forth. However, it is understood that 
embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, 
structures and techniques have not been shown In detail in order not to obscure an understanding of this description. 
[0057] The following terms are provided solely to aid in the understanding of the invention. 

40 [0058] In the following description the pixels or sub-pixels that are used to mask the defective pixel are called "masking 
elements" and the defective pixel or sub-pixel itself is called "the defect". By a defective pixel or sub-pixel is meant a 
pixel that always shows the same luminance, i.e. a pixel or sub-pixel stuck in a specific state (for instance, but not limited 
to, always black, or always full white) and/or color behavior independent of the drive stimulus applied to it, or a pixel or 
sub-pixel that shows a luminance or color behavior that shows a severe distortion compared to non-defective pixels or 

45 sub-pixels of the display. For example a pixel that reacts to an applied drive signal, but that has a luminance behavior 
that is very different from the luminance behavior of neighboring pixels, for instance significantly more dark or bright 
than surrounding pixels, can be considered a defective pixel. By visually masking Is meant minimizing the visibility and 
negative effects of the defect for the user of the display. 

[0059] In the present description, the terms "horizontal" and "vertical" are used to provide a co-ordinate system and 
50 for ease of explanation only. They refer to a co-ordinate system with two orthogonal directions which are conveniently 
referred to as vertical and horizontal directions. They do not need to, but may, refer to an actual physical direction of the 
device. In particular, horizontal and vertical are equivalent and interchangeable by means of a simple rotation through 
an odd multiple of 90°. 

[0060] A matrix addressed display comprises Individual display elements. The display elements, either themselves 
55 or in groupings, are individually addressable to thereby display or project an arbitrary Image. In the present description, 
the term "display elements" Is to be understood to comprise any form of element which modulates a light output, e.g. 
elements which emit light or through which light is passed or from which light is reflected. The term "display" includes a 
projector. A display element may therefore be an individually addressable element of an emissive, transmissive, reflective 
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or trans-reflective display, especially a fixed format display. The term "fixed format" relates to the fact that an area of 
any image to be displayed or projected is associated with a certain portion of the display or projector, e.g. in a one-to- 
one relationship. Display elements may be pixels, e.g. in a grayscale LCD, as well as sub-pixels, a plurality of sub-pixels 
forming one pixel. For example three sub-pixels with a different color, such as a red sub-pixel, a green sub-pixel and a 

5 blue sub-pixel, may together from one pixel in a color display such as an LCD. Whenever the word pixel is used, it Is to 
be understood that the same may hold for sub-pixels, unless the contrary is explicitly mentioned. 
[0061] The invention will now be described by a detailed description of several embodiments of the Invention. It is 
clear that other embodiments of the invention can be configured according to the knowledge of persons skilled in the 
art without departing from the true spirit or technical teaching of the invention, the invention being limited only by the 

10 terms of the appended claims. 

[0062] The disclosure includes a mathematical model that may be used to calculate the optimal driving signal for the 
masking elements In order to minimize the visibility of the defect(s). The same algorithm can be adapted for use with 
different display configurations because it includes some parameters that describe display characteristics. A mathemat- 
ical model based on the characteristics of the human eye is used to calculate the optimal drive signals of the masking 

15 elements. The model describes algorithms to calculate the actual response of the human eye to the superposition of 
the stimulus applied (in casu to the defect and to the masking pixels). In this way the optimal drive signals of the masking 
elements can be described as a mathematical minimization problem of a function with one or more variables. It Is possible 
to add one or more boundary conditions to this minimization problem. Examples when extra boundary conditions may 
be desirable are in case of defects of one or more masking elements, limitations to the possible drive signal of the 

20 masking elements, dependencies In the drive signals of masking elements, etc. 

[0063] While methods and systems according to embodiments of the present Invention described herein do not repair 
the defective pixels, in many cases they may be applied to make the defects less visible, perhaps nearly invisible, and 
thus may help to avoid an incorrect image interpretation. 

[0064] While certain inventive principles are described with reference to flat panel displays, the scope of the invention 
25 Is not limited thereto. It is also understood that a flat panel display does not have to be exactly flat but includes shaped, 
bent or bendable panels. A flat panel display differs from a display such as a cathode ray tube in that it comprises a 
matrix or array of "cells" or "pixels" each producing or controlling light over a small area. Arrays of this kind are called 
fixed format arrays. There is a relationship between the pixel of an image to be displayed and a cell of the display. Usually 
this is a one-to-one relationship. In some cases, each cell may be addressed and driven separately. Applicability of the 
30 general inventive principles Is not limited to whether the flat panel displays are active or passive matrix devices. The 
array of cells is usually in rows and columns but the present invention is not limited thereto but may include any arrange- 
ment, e.g. polar or hexagonal inventive principles will mainly be described with respect to liquid crystal displays but in 
general such principles are more widely applicable to flat panel displays of different types, such as plasma displays, 
field emission displays, EL-displays, OLED displays etc. In particular such principles may be applied not only to displays 
35 having an array of light emitting elements but also displays having arrays of light emitting devices, whereby each device 
is made up of a number of Individual elements. The displays may be emissive, transmisslve, reflective, or trans-reflective 
displays. 

[0065] Further the method of addressing and driving the pixel elements of an array is not considered a limitation on 

the general applicability of the inventive principles. Typically, each pixel element is addressed by means of wiring but 
40 other methods are known and are useful with the invention, e.g. plasma discharge addressing (as disclosed in US Pat. 
No. 6,089,739) or CRT addressing. 

[0066] A matrix addressed display 12 comprises individual pixels 14. These pixels 14 can take all kinds of shapes, 
e.g. they can take the forms of characters. The examples of matrix displays 1 2 given in FIG. la to FIG.2b have rectangular 
or square pixels 14 arranged in horizontal rows and vertical columns. FIG. 1 a illustrates an image of a perfect display 

45 12 having equal luminance response in all pixels 14 when equally driven. Every pixel 14 driven with the same signal 
renders the same luminance. In contrast, FIG. 1 b illustrates an image of a display 12 where the pixels 14 of the display 
1 2 are also driven by equal signals, but where the pixels 1 4 render a different luminance, as can be seen by the different 
grey values. Pixel 1 6 in the display 1 2 of FIG. 1 b is a defective pixel. FIG. 1 b shows a monochrome pixel structure with 
one defective pixel 16 that is always in an intermediate pixel state. 

50 [0067] FIG. 2a shows a typical RGB-stripe pixel arrangement of a color LCD display 12: one pixel 1 4 consists of three 
colored sub-pixels 20, 21, 22 In stripe ordering. These three sub-pixels 20, 21, 22 are driven Individually to generate 
color images. In FIG. 2a there are two defective sub-pixels present: a defective red sub-pixel 24 that is always off and 
a defective green sub-pixel 25 that is always fully on. In the present examples, the different coloured subpixels are 
arranged so that columns of sub-pixels In the display have the same colour. The colour is indicated for each column at 

55 the bottom of the drawing. 

[0068] FIG. 2b shows an asymmetric pixel structure that is often used for high-resolution monochrome displays. In 
FIG. 2b, one monochrome pixel 14 consists of three monochrome sub-pixels. Depending on the panel type and driving 
electronics the three sub-pixels of one pixel are driven as a unit or individually. FIG. 2b shows 3 pixel defects: a complete 
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defective pixel 1 6 in "always on" state and two defective sub-pixels 27, 28 in "always off' state that happen to be located 
in a sanne pixel 14. 

[0069] The spatial distribution of the luminance differences of the pixels 14 can be arbitrary. It is also found that with 
many technologies, this distribution changes as function of the applied drive to the pixels indicating different response 
relationships for the pixels 14. For a low drive signal leading to low luminance, the spatial distribution pattern can differ 
from the pattern at higher driving signal. 

[0070] The optical system of the eye, in particular of the human eye, comprises three main components: the cornea, 
the iris and the lens. The cornea is the transparent outer surface of the eye. The pupil limits the amount of light that 
reaches the retina and it changes the numerical aperture of the optical system of the eye. By applying tension to the 

lens, the eye is able to focus on both nearby and far away objects. The optical system of the eye is very complex but 
the process of image transmission can be approximated by using a "black-box" approach. The behavior of the black 
box can be described by the complex pupil function CPF: 

CPF{x, y) = P{x,y) • exp[- ikW{x,y)\ 

where / denotes k = 2k fk\s the wave number, and X is the wavelength of the light. The complex pupil function 
includes two components: an amplitude component P(x,y) which defines the shape, size and transmission of the black 
box; and a phase component including a wave aberration W(x,y) which defines how the phase of the light has changed 
after passing through the black box. Other expressions of the complex pupil function are also known, and the range of 
embodiments is not limited by any particular such expression. 

[0071] It is common practice in vision applications to describe the wave aberration W(x,y) mathematically by means 
of a set of polynomials. Often Seidel polynomials are used, but also Taylor polynomials and Zernike polynomials are 
common choices. Especially Zernike polynomials have interesting properties that make wave aberration analysis much 
easier. Often unknown wave aberrations are approximated by Zernike polynomials; the coefficients of the polynomials 
are typically determined by performing a least-square fit. 

[0072] Once the nature of the light transmitted by the eye's optical system is known or modeled, the image formation 
process can be described by a transfer function that models a response of the human visual system (HVS). For example, 
the transfer function may model the projection of a given visual stimulus on the retina. Most lenses, including the human 
lens, are not perfect optical systems. As a result when visual stimuli are passed through the cornea and lens the stimuli 
undergo a certain degree of degradation or distortion. 

[0073] Transfer functions that may be used to model the HVS response include the 'Pupil Function (or aberration)', 
the 'Point Spread Function (PSF)', the 'Line Spread Function (LSF)', the 'Contrast Transfer Function (CTF)', the 'Optical 
Transfer Function (OTF)', the 'Modulation Transfer Function (MTF)' and 'Phase Transfer Function (PTF)'. Clear math- 
ematical relations exist between all these representation-methods so that it is possible to transform one form into another 
form. For example, the OTF is the Fourier transform of the PSF, and it is also the product of the MTF and PTF. Expression 
of such a modeling transfer function may be done analytically (for instance but not limited to a mathematical function in 
Cartesian or polar co-ordinates, by means of standard polynomials, or by means of any other suitable analytical method) 
or numerically by describing the function value at certain points. For convenience, use of the PSF is described below, 
but it is expressly contemplated and hereby disclosed that any of the transfer functions identified above may be used, 
as well as any other model of the HVS response. 

[0074] The degradation or distortion of the visual stimuli can be represented by projecting an exceedingly small dot 
of light, a point, through a lens. The image of this point will not be the same as the original because the lens will introduce 
a small amount of blur. The PSF describes the image of a point source formed by the black box. The PSF of the eye 
can be calculated using the Fraunhofer approximation: 

PSF{x',y') = K*\F{CPF{x,y)f 

where (V,/) denotes a coordinate system of the image plane, (x,y) denotes a coordinate system of the object plane, F 
denotes the two-dimensional Fourier transform, K is a constant, and I I represents the magnitude-operator. The image- 
plane and object-plane coordinate systems are related as (x\ /) = (Gx,Gy), where Mis a magnification constant that 
depends upon, for example, the object distance. Other expressions of the PSF are also known, and the range of em- 
bodiments is not limited by any particular such expression. 

[0075] In the case of the human eye, the PSF describes the image of a point source on the retina. To describe a 
complete object one can think of an object as a combination or a matrix of (a potentially exceedingly large number or 
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infinite number of) point sources. Each of these point sources is then projected on the retina as described by the same 
PSF (this approximation is strictly only valid If the object is small and composed of a single wavelength). Mathematically 
this can be described by means of a convolution: 



where 0(>^, j/) is the object representation at the image-plane and l(x', y) is the resulting image on the retina. Typically 
this convolution will be computed in the Fourier domain by multiplying the Fourier transforms of both the PSF and the 
object and then applying the inverse Fourier transform to the result. 

[0076] FIG. 3a shows an analytical PSF in case the optics is considered to be diffraction-limited only. It is to be noted 
that the PSF is clearly not a single point, i.e. the image of a point source is not a point. The central zone of the diffraction- 
limited PSF is called an Airy disc. FIG. 3b and 3c show (numerical) PSFs that were measured on test subjects. Here 
again it can be seen that the PSF is not a point. 

[0077] Based on the PSF or other model of the HVS response, the response or expected response of the eye to a 
defective pixel can be mathematically described. Therefore the defective pixel is treated as a point source with an "error 
luminance" value dependent on the defect itself and the image data that should be displayed at the defect location at 
that time. For instance if the defective pixel is driven to have luminance value 23 but due to the defect it outputs luminance 
value 3, then this defect is treated as a point source with error luminance value -20. It is to be noted that this error 
luminance value can have both a positive and a negative value. Supposing that some time later this same defective 
pixel is driven to show luminance value 1 but due to the defect it still shows luminance value 3, then this same defective 
pixel will be treated as a point source with error luminance value +2. 

[0078] As described above, this point source with a specific error luminance value will result in a response of the eye 
as described by the PSF or other model of the HVS response. Because this response is typically not a single point, it is 
possible to use pixels and/or subpixels in the neighborhood of the defective pixel to provide some image improvement. 
These neighboring pixels are called masking pixels and can be driven in such a way as to minimize the response of the 
eye to the defective pixel. This effect may be achieved by changing the drive signal of the masking pixels such that the 
superposition of the image of the masking pixels and the image of the defective pixel results in a lower or minimal 
response of the human eye. Calculation of the masking values to be superposed may be expressed by a minimization 
operation such as the following: 



where Ci,C2,...,C„ are the masking luminance values to be superposed on the masking pixels Mi,M2,...,MnWith relative 
locations (x^,y^)Xx2,y2),---,(Xn,yn) '"^ order to obtain minimal eye response to the defect. The cost function f(v,x\ y') 
calculates a "penalty" value from the eye response at location (x",/). Examples of / (v, x', y') include v^, Ivl, and v2/V(x') 
2+(y')2 , although the cost function f is not limited to these examples. 

[0079] It is to be noted that the Cartesian coordinate system (y, y) is defined in the image plane on the retina, with 
origin being the centre of the image of the defect as described by the model of the HVS response (e.g., the center of 
the PSF of the defect PSF(x\ y")). As noted above, the Cartesian co-ordinate system (x,y) is defined in the object plane 
of the display, and (X/,y^) denotes the location of masking pixel / relative to the defect. The relation between these two 
co-ordinate systems may be expressed as (x',y") = (Gx,Gy) , where G is a constant that defines the magnification in the 
image plane and whose value may depend on, among other factors, the object distance. 

[0080] Various forms may be adopted for the superposition function v. For a case in which the neighborhood includes 
only one defect to be masked, the function vmay be expressed as 



where E indicates the error luminance value of the defect, (x',y) indicates the location of the image of the defect, n 
indicates the number of masking pixels, and (V/,//) indicates the location of the image of masking pixel /. 



I{x',y') = PSF®Oix',y'), 




(1) 




(2) 
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[0081 ] In another case, nnultiple defects nnay occur within a snnall area, the small area being the area that contains all 
nnasking pixels for one particular defect. In this case it nnight not be possible to assign the required value to all nnasking 
pixels. In one such example, the mathematical description is restated such that one of the defects is chosen as the 
centre of both the image plane and object plane co-ordinate systems. The algorithm may then be arranged to minimize 
the total response to all the defects and all used masking pixels in this area. For example, the superposition function y 
may be expressed as 



V = 



g C, X PSF{x' - x\, y' - y\-^ + x PSF{x' ,y') + 



^Ej^PSF{x'-ex),y'-ey'j) 



(3) 



where the selected defect 1 is at location (x',y'), m indicates the number of defects, Ey indicates the error luminance 
value of defect y, and (eXj',ey'j) indicates the location of the image of defect j. 

[0082] As the response of each HVS may be different, defect compensation and/or masking according to a method 
as described herein can be made user specific by using one or more response models (e.g., PSFs) which are specific 
for that user. 

[0083] FIG. 4a shows the eye response to a single defective pixel in the image plane if no masking is applied. FIG. 
4b shows the eye response to the same defective pixel but after masking using 24 masking pixels (neighbours of the 
defective pixel) has been applied. FIG. 4c shows the centre locations of the PSFs in the image plane of the masking 
pixels and the defective pixel (central point). These simulations have been performed with the diffraction limited PSF 
and the minimization was done numerically by using a least square error method. 
[0084] The PSF of a diffraction limited optical system is given by (in polar coordinates): 



PSF{r') = 2 



where J1 is the Bessel function of the first kind and r' is given by 

r = •r , 

¥ 

where D is the aperture diameter, f is the focal length, and X is the wavelength of the light. This expression implies that 
the exact PSF is dependent on the iris diameter of the eye. Therefore, an improvement could be to adapt the PSF (and/or 
other model of the HVS response) used for the minimization calculation based on the average luminance value of the 
display or some part of the display such as the neighborhood of the defect and/or the average luminance value of the 
environment. 

[0085] To simplify the calculation, some changes to the algorithm can be made. A first possible change is to restrict 
the integration in Eq. 1 to a limited area around the defect. This is possible because the result of the cost function (and 
the value of an HVS response model, such as the PSF) typically decreases very fast with increasing distance from the 
defect. If symmetric models (e.g., symmetric PSFs) are used or if the pixel structure is symmetrical, then it is often 
possible to apply some boundary conditions to the masking values of the masking pixels. For example, in case of a point 
symmetric pixel structure, a point-symmetric PSF may be used such that the calculated masking values for the masking 
pixels will show point symmetry also. 

[0086] Another possible change is to approximate the integration over a certain area as a summation over particular 
points in that area. Such a technique is generally used in mathematics. If calculation time is very important, then the 
two-dimensional minimization problem can betransformed or approximated into a one-dimensional problem (for example, 
by transforming or approximating a two-dimensional model such as PSF(x\ y') by a one-dimensional version such as 
PSF(0). 

[0087] In general, the inventive principles are not limited to any particular coordinate system such as the Cartesian 
co-ordinate system as used above; other systems are also possible, for instance, but not limited to, a polar coordinate 
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system. 

[0088] As described by way of example above, the problem of finding an optimal masking luminance of the masking 
pixels may be translated into a well-understood minimization problem. It is to be noted that this mathematical description 
is very general: it does not impose any limitation on the number of masking pixels nor on the location of these masking 
pixels. The pixels also do not need to be located in any particular pixel structure: the algorithm can handle all possible 
pixel organizations. Also the defect itself is not necessarily located at a pixel location: for example some dust between 
two pixels can cause a permanent bright spot. 

[0089] The algorithm above describes a general method to calculate optimal luminance values for masking pixels in 
order to minimize the eye response to the defect. In practice, however, some special situations exist for which additions 

to the described algorithm may be desirable. 

[0090] A first special situation is when the pixels cannot be driven individually, but are rather driven in groups. High- 
resolution monochrome LCDs, for example, often have a pixel structure where one monochrome pixel consists of three 
monochrome sub-pixels that are equally and simultaneously driven, as illustrated in FIG. 2b. In such a situation it may 
be desirable to apply a boundary condition to the minimization problem to be solved, in order to respect this driving 
method. In the case of three equally and simultaneously driven sub-pixels, for example, it may be desirable for the 
boundary condition to state that the masking values of each of the simultaneously driven subpixels within a same pixel 
should have a same value. 

[0091] Asecond special situation occurs when pixels have a limited driving range. It is possible thatthe above-described 
minimization algorithm would result in a required luminance value for a masking pixel that lies outside of the luminance 
range of the pixel. Introducing a boundary condition that limits the driving value of all pixels solves this problem of an 
invalid value. Such type of boundary condition can be stated as: 



LL <= (Pixel value + masking value) <= UL 

and this for all masking pixels. In this expression LL is the lower driving limit of the pixels and UL is the upper driving 
limit. "Pixel value" is the normal (uncompensated) pixel value of the pixel and "masking value" is the calculated value 
to be superposed on that masking pixel. 

[0092] Furthermore, the requirement that the final driving value of the masking pixel should be an integer can be a 

boundary condition to be used. 

[0093] Another special situation occurs when pixels (or defects) are larger so that they cannot be modeled anymore 
by a point source. One potential solution is to model the defect as a (possibly infinite) number of point sources. An 
example could be a dual domain in-plane switching (IPS) LCD panel where pixels consist of two domains. Such pixels 
can be modeled by two or more point sources that do not have necessarily the same luminance value. FIG. 5a shows 
nine pixels 50 each having three sub-pixels 51 and each sub-pixel 51 having two domains 52, 53. FIG. 5b shows one 
pixel 50 in detail. In this situation it may be desirable to treat each pixel 50 as a superposition of 6 point sources. If the 
pixel 50 can only be driven as a unit, it may be desirable to add a boundary condition stating that the 6 masking values 
of each pixel 50 should be equal. 

[0094] The algorithms described use luminance values and not driving values. Typical displays however have no 
linear relation between driving level of a pixel and resulting luminance value. Therefore, in a realistic display system, 
the calculated luminance adjustment should be transformed into a required drive level adjustment. Typically a display 
system has one or more look-up tables (LUTs) connected to a panel with a specific gamma curve. The conversion from 
luminance value to driving value is straightforward by applying the inverse operations. It is to be noted that depending 
on the exact location where the compensation and/or masking will be applied, the LUT inversion may or may not be 
necessary. FIG. 6 shows a typical transformation from driving level to the resulting luminance level. In cases where the 
conversion between driving level of a pixel and resulting luminance value may be modeled adequately, the algorithms 
(e.g., expression (1)) may be configured to include such conversion. Conversion of a driving level to an intermediate 
value using a first look up table (LUT) may for example be performed, as shown in FIG. 6, and further conversion of 
intermediate values using further look up tables (LUT) may be performed resulting in a luminance value after applying 
a panel gamma curve. In other words, conversion from the driving level to the luminance value is in the present example 
performed using look up tables and using the panel gamma curve. 

[0095] The examples described above relate to monochrome displays. For defect compensation and/or masking in 
color displays, any of the following three extensions may be used. FIG. 9 shows eight different examples of subpixel 
geometries that may be used in a color display panel, where R,G,B,W,C,M,Y denote red, green, blue, white, cyan, 
magenta, and yellow. 

[0096] A first method is to use only masking sub-pixels of the same color as the defective sub-pixel. This method is 
simple, but can introduce visible color shifts since the color value of the defective pixel and the masking pixels can change. 
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[0097] Therefore, a second method is proposed, according to which artificial defects are introduced such that the color 
points or color co-ordinates of the defective pixel and the nnasking pixels change only a little or do not change at all. For 
exannple, suppose that In a color panel with RGB sub-pixels a particular R sub-pixel is defective such that the color point 
of that pixel is incorrect. Such a method may be applied to introduce an artificial G- and/or B-defective sub-pixel such 

5 that the color point or color coordinates of the defective pixel remain correct as much as possible, although the luminance 
value may be incorrect. It is to be noted that it is not always possible to correct the color point completely with the 
remaining sub-pixels. To restate this method, the drive values of the two remaining non-defective sub-pixels may be 
changed so that the color point of the pixel as a unit remains as close to the correct value as possible. It will be obvious 
for those skilled in the art that this is easy to perform once the (Y,x,y) co-ordinates of each sub pixel type (for example 

10 red, green and blue sub-pixels in case of a color display as in FIG. 2a) are available. These (Y,x,y) co-ordinates, where 
Y is the intensity and x,y are the chromaticity co-ordinates, can be measured easily for each of the subpixel types and 
at one or more drive levels. The masking pixels are then calculated with the normal minimization problem for each color 
independently where the artificial defects are treated as real defects. 

[0098] It is known that the human eye is more sensitive to intensity differences than to chromaticity differences. 

15 Therefore a third method permits a color point error in order to keep the intensity error due to the defect as small as 
possible. Such an effect may be achieved by minimizing principally (or only) the intensity response of the eye. In one 
example, the drive signals for driving the remaining non-defective sub-pixels are changed in such a way that the luminance 
intensity error of the pixel as a unit is as small as possible, while the color of the pixel as a unit may deviate from the 
color originally intended to be displayed. This change is again easy to perform once the (Y,x,y) co-ordinates of each 

20 sub-pixel type (for example red, green and blue sub-pixels in case of a color display as in FIG. 2a) are available. Also 
in this case virtual defects may be introduced, possibly making the chromaticity error larger while minimizing the intensity 
error. It is for example known that red and blue sub-pixels have a smaller intensity value than a green sub-pixel at a 
same level of a drive signal. If a green sub-pixel is defective, therefore, such a method may be configured to drive the 
red and blue sub-pixels so as to have a higher intensity level. 

25 [0099] Of course, it is also possible to mix the three methods described above. This can be favorable for instance if 
the goal would be to limit at the same time both the intensity and color temperature errors with one of them possibly 
being more important than the other. 

[0100] It is to be noted that typically a model of the HVS response (such as the PSF) is (slightly) wavelength dependent. 
So different models (e.g., PSFs) can be used for each sub-pixel color. FIG. 7a shows a real green defective subpixel 

30 71 present in the display 70. FIG. 7b shows the same green defective sub-pixel 70 and artificial red and blue defective 
sub-pixels 72, 73 introduced to retain the correct color co-ordinate of the pixel. The artificial defective pixels 72, 73 are 
not really present in the display but are introduced by altering the driving level of these pixels. For the situation in FIG. 
7b, the minimization problem may be solved based on three defective sub-pixels: one really defective sub-pixel 71 and 
two artificially introduced defective sub-pixels 72, 73. For a case in which a pixel or subpixel has different PSFs for 

35 different color channels, expression (1) may be extended as follows: 



[C,,C2,...,CJ = argmin f {f(,v^,V2,...,v„,x',y')dx'dy', 
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where v ^ = E^PSF^^^' ,y') + 



Y,C,^PSF^ix'-x\,y'-y\) 



Li=i 



for 1 < j < m (for a three-channel color 



scheme such as RGB or YUV, m = 3), PSFj^ indicates the PSF of the defect that contributes to color channel j, and 
PSFy^- indicates the PSF of the /-th masking pixel or subpixel that contributes to color channel j. Such an expression may 
also be applied to an RGBW display having white subpixels. In this case, the white subpixel would have a nonzero PSF 
for all three color channels, while subpixels of the individual colors may have nonzero contributions for only one color 
(if the subpixel is a pure color) or for more than one color (if the subpixel is not perfectly limited to only one color 
component). Examples of f(v^,V2,...,v^,x\ y) include {Zvf , Zv^^- , \i:Vj\, Zv2y^(y)2 +(y)2 , and (Ev-)2/V(y)2 +(y)2 for 1 < 
J< m, although the cost function f is not limited to these examples. In further examples, cost function f applies different 
weights to the contributions of the various color channels and/or weights luminance error more heavily than chrominance 
error. The cost function f may also be configured according to more complex models of the HVS to account for a contrast 
sensitivity function), effects of masking and/or lateral inhibition, and/or similar artifacts of neuronal processing. 
[0101] Visual compensation and/or masking of a defect as described herein can be done both in software and in 
hardware. Such an operation transforms the image into a pre-corrected image based on any of the schemes as described 
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herein. Some possible implementations of where the defect compensation and/or masking can be done are shown in 
FIG. 8, which illustrates possible locations for a real-time defect processing system. A host computer 1 10 and display 
120 are shown by way of example in FIG. 8. As illustrated by refence number 101, the defect compensation and/or 
masking may be done by the CPU 1 12 of the host computer 110, for instance in the driver code of the graphical card 
or with a specific application or embedded in a viewing application. Alternatively, as illustrated by reference numbers 
1 02 and 1 03, the defect compensation and/or masking may be done in the graphical card 1 1 4 of the host computer 1 1 0, 
either in hardware 1 16 or in fimnware 118. According to still another embodiment, as illustrated by reference numbers 
1 04 and 1 05, defect compensation and/or masking may be done in the display 1 20, either in hardware 1 22 or in firmware 
124. And according to yet another embodiment, as illustrated by reference number 106, defect compensation and/or 
masking may be done on the signal transmitted between the graphical card and the display, anywhere in the datapath 1 30. 
[0102] It is to be noted that defect compensation and/or masking methods as described herein may be implemented 
for execution in real-time (at least at the frame rate of the display) and/or off-line (once, at specific times or at a frame 
rate lower than the display frame rate). 

[0103] Application of a defect compensation and/or masking method as described herein may help to avoid that a 
user of the display mistakes the defective pixel for a real signal present in the displayed image. In case of radiology, for 
example, it is possible that a radiologist would treat the defect as really present, which could reduce the quality of a 
diagnosis. In another situation, application of such a method may help to avoid frustration of the user because his/her 
possibly new display shows one or more extremely visible pixel defects. 

[0104] A device configured to characterize pixels of a display comprises a vision measurement system, a set-up for 
automated, electronic vision of the individual pixels of the matrix addressed display, i.e. for measuring the light output, 
e.g. luminance, emitted or reflected (depending on the type of display) by individual pixels 14. The vision measurement 
system comprises an image capturing device, such as for example a flat bed scanner or a high resolution CCD camera, 
and possibly a movement device for moving the image capturing device and the display 12 with respect to each other. 
The image capturing device generates an output file, which is an electronic image file giving a detailed picture of the 
pixels 14 of the complete electronic display 12. Once an image of the pixels 14 of the display 12 has been obtained, a 
process is run to extract pixel characterization data from the electronic image obtained from the image capturing device. 
[0105] Instead of luminance, also color can be measured. The vision set-up is then slightly different, and comprises 
a color measurement device, such as a colorimetric camera or a scanning spectrograph for example. The underlying 
principle, however, is the same: a location of the pixel and its color are determined. 

[0106] As described above, a defect in a display panel may be masked by applying a cost function, based on a model 
that represents a response of a human visual system, over a neighborhood of the defect. Such a method may include 
masking one or more defects by modulating other pixels in the neighborhood to minimize the cost function. 
[0107] Such a method may include minimizing a cost function with respect to a projection of the defect on the retina. 
In addition to the response of the optical system of the eye, however, other effects such as neuronal processing may 
limit perception. When viewing a bright edge, for example, a HVS will be less sensitive to small changes in image content 
that are close to the edge. This effect is one manifestation of a mechanism called lateral inhibition. A HVS is also less 
sensitive to high frequencies at low amplitudes, an effect described by a contrast sensitivity function (CSF), which is 
typically higher for intensity than for color and may also differ between colors. As a consequence of such processing, a 
defect may be rendered invisible even if the projection on the retina is not perfect. It may be desirable to take account 
of such effects such that compensation and/or masking of defects that are beyond the limits of perception may be reduced 
or avoided. Alternatively or additionally, therefore, a method of defect compensation and/or masking may include min- 
imizing a cost function with respect to post-processing of the retinal image by neurons (e.g., in the eye and/or in the brain). 
[0108] A contrast sensitivity function (CSF) describes, for each of a range of spatial frequencies measured in cycles 
per degree, the minimum amplitude needed for a sinusoid of that frequency to be perceptible. It may be desirable to 
select a typical viewing distance in order to provide a relation between cycles per degree and distance along a plane of 
a panel. A minimization algorithm according to expression (1 ) may also be modified according to other simple or complex 
models of the HVS, such as achromatic models proposed by Movshon (CSF(f) =aPe-^^, where ^denotes frequency in 
cycles per degree and values of 75, 0.2, and 0.8 may be used for a, b, and c in general use), P.G.J. Barten (e.g., in 
Contrast Sensitivity of the Human Eye and its Effects on Image Quality, SPIE, Bellingham, WA, 1999), and S. Daly (e.g., 
in "Subroutine for the generation of a two dimensional human visual contrast sensitivity," Technical Report 233203Y, 
Eastman Kodak, 1987) and/or chromatic models proposed by K.T. Mullen (e.g., in "The contrast sensitivity of human 
colour vision to red-green and blue-yellow chromatic gratings," Journal of Physiology, 359: 381-400, 1985), S. Daly (e.g., 
in chap. 13 ("The Visible Differences Predictor: An algorithm for the assessment of image fidelity") of Digital Images and 
Human Vision, MIT Press, Cambridge, MA, 1993), and S.N. Pattanaiket al. (e.g., in "Multiscale model of adaptation, 
spatial vision and colour appearance," Proc. SIGGRAPH '98, pp. 287-88, July 1998). 

[0109] Another cost function minimization procedure that may be used for defect masking as disclosed herein is 
described in U.S. Publ. Pat. Appl. 2005/01 69551 (Messing et al.) and papers 1 5.2 ("Optimal Rendering for Colour Matrix 
Displays," L.J. Kerofsky and D.S. Messing) and 15.3 ("An Application of Optimal Rendering to Visually Mask Defective 
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Subpixels," D.S. Messing and L.J. Kerofsky) of the Proceedings of ADEAC2005 (Society for I nfornnation Display, October 
24-27, 2005). Such a procedure includes formulating a set of constraints (e.g., Lagrange constraints) to nnodel a given 
subpixel geonnetry. The procedure also includes fornning an error nneasure. A difference is calculated between the actual 
panel and an ideal display in a 1-D or 2-D spatial domain, and this difference is transfonned into a frequency-domain 
array Ef. The array Ef is perceptually weighted by applying a set of frequency-weighting filters that model the human 
visual system's sensitivity to luminance and chrominance detail. The error measure is calculated as a sum of the mag- 
nitudes (e.g., as a sum of the L2 nonns) of the weighted components of Ef. An optimization problem based on the error 
measure and the set of constraints (e.g., a constrained Lagrange optimization problem) Is solved to obtain an array of 
shift-variant filters, and these filters are applied to the respective color channels of the signal to be displayed. 
[0110] An implementation of a defect compensation and/or masking method as described herein may be adapted for 
use with a high dynamic range (HDR) display. One type of HDR display is a multi-panel HDR display, which includes a 
stack of two or more panels. 

[0111] According to one aspect, the present invention relates to a display assembly comprising a first imaging layer 
and a second imaging layer. The display assembly may for example be a multi-panel display assembly, including a stack 
of two or more panels. The first Imaging layer and the second imaging layer may be provided in a display panel and a 
second display panel. In some embodiments of the present aspect, a drive signal to a pixel on one of the first and second 
Imaging layers in the display assembly is based on a defect in the other of the first and second Imaging layers. The 
multi-panel display assembly may comprise imaging layers and/or display panels based on any suitable display tech- 
nology as described above for the single display panel or imaging layer, such as for example for fixed format display 
technologies, active or passive matrix display technologies, emissive, reflective, transmlssive or trans-reflective display 
technologies, display technologies using a backlight, display technologies based on liquid crystal technology, plasma 
technology, field emission display technology, EL technology, OLED technology, digital mirror device technology etc. It 
is possible that some of the components that are customary in a single panel, as for example described above, can be 
omitted or removed when two or more panels are stacked, depending on the display technology used. If for example a 
display technique using polarisation Is used, in a two-panel stack, for example, the back polarizer of the front panel may 
be omitted or removed, since the front polarizer of the back panel can serve this purpose. Another example is that certain 
foils, such as brightness enhancement foils and polarization recycling foils, if present, may only be applied in the back 
panel. If backlight technology is used, of course, only one backlight is necessary. Such a display may achieve a contrast 
range that Is theoretically the product of the contrast ranges of the individual panels. For example, two panels each 
having a contrast range of 100:1 may be stacked to achieve a theoretical contrast range of 10,000:1. Panels having 
contrast ranges of up to 1000:1 are presently available commercially, such that a multi-panel display having a contrast 
range of 1,000,000:1 may be currently possible. Because the maximum transmittance of each panel is less than 100%, 
possibly much less (typically 1 0% maximum), a multi-panel display may require a much stronger backlight than a single- 
panel display. 

[01 1 2] According to some embodiments of the present aspect, a drive signal to a pixel on one of the first and second 
imaging layers in the display assembly is based on a defect in the other of the first and second imaging layers. Note 
that a defect, e.g. a defective pixel, is not limited to a completely black or completely white pixel. If a single pixel is 
defective In a multi-panel display, for example, then the defective pixel (which Is viewed as the combination of the pixels 
of several panels that are on top of each other) could be perceived as a pixel having a limited or reduced dynamic range 
or luminance range as compared to non-defective pixels. Even in a case where the defective pixel is stuck at completely 
black or white, the variable transmittance of other pixels in the optical path by which light passing through that pixel 
reaches the viewer's eye may result in an overall perception of a limited or reduced dynamic range or luminance range 
rather than a perception of an entirely white or black pixel. It is also possible that only one domain of a pixel is defective. 
In some panels, a pixel is split into multiple spatial domains such that sub-pixels of one domain appear brighter when 
viewed from one off-axis direction (e.g., from the left) and sub-pixels of another domain appear brighter when viewed 
from another off-axis direction (e.g., from the right). 

[0113] Masking values may be applied in a neighbourhood that extends in three dimensions: the two dimensions of 
the plane of the panel having the defect, and another dimension to include at least one pixel of another panel in the 
stack. For example, such a masking may be applied over a neighborhood that is square in at least one panel (e.g., a 
3x3 neighborhood in the plane of the panel, centered at the defect) and/or a neighborhood of some other shape (e.g., 
circular or diamond-shaped). A minimization algorithm as expressed In equation 1 may be extended to apply to a 
neighbourhood having three spatial dimensions. For example, such an algorithm may be modified to include a common 
luminance and/or color offset for two or more of the defective and/or masking pixels, where the common offset is due 
to a larger pixel In a corresponding area of another panel. It Is also possible that a pixel of a panel can also Influence 
parts of one or more pixels of one or more panels In the other layers. If the backlight Is not perfectly colllmated, then 
one pixel may Influence more than one pixel above it, even If the respective panels have the same resolution and are 
perfectly aligned. In such a case, the radiation pattern from the pixel may have a distribution that is somewhat Lambertian. 
[0114] In some cases, it may be desirable to restrict the neighborhood in the plane of the pixel to one line (e.g., the 
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line containing the defect). In other cases, it nnay be desirable to restrict the neighborhood to all pixels above or below 
the defect (e.g., pixels belonging to different panels), which is to say a line containing the defect that is perpendicular to 
the display plane of the panel. In further cases, it nnay be desirable to restrict the neighborhood to the line containing 
the defect and all pixels above or below the defect. Such restrictions may require less hardware to implement. For 
example, additional storage (e.g., line buffers and/or delay elements) may be needed to implement correction over two 
dimensions in the display plane of the panel, and a restriction to one dimension in the display plane of the panel may 
allow a reduction In such hardware. It is also possible to apply such a restriction to a defect compensation and/or masking 
method as applied to a single-panel display. 

[0115] Application of defect compensation and/or masking to a multi-panel display typically offers more degrees of 
freedom in compensating and/or masking defects. For example, such a method may be configured to modulate a 3-D 
neighborhood of the defect. Such a masking method may be generally characterized as applying a filter to the masking 
pixels, where the filter also depends on characteristics of at least one other layer of the multi-panel display. Alternatively, 
such a method may be generally characterized as applying a filter that is dependent on an image contents of another 
panel. Methods of defect compensation with or without masking are also described herein. 

[0116] The panels of a multi-panel display may have the same resolution. However, an extreme contrast range is not 
typically needed over a small area (e.g., pixel by pixel), and high-contrast modulation is usually applied at lower frequen- 
cies. In some cases, it may be desirable to implement a multi-panel display using panels of different resolutions. For 
example, a back panel of lower resolution, which will typically be less expensive, may be used to display low-frequency 
information. A two-panel stack may include a front panel having a standard resolution size of 1600x1200 pixels and a 
back panel having a standard resolution size of 1 280x1 024 or 1 024x768 pixels, although any other standard or non- 
standard resolutions may also be used. In another example, the number of pixels in one or both directions of a high- 
resolution panel is an integer multiple of the number of pixels in the same direction of a low-resolution panel. For example, 
a front panel having a resolution of 1 600x1 200 may be paired with a back panel having a resolution of 800x600, so that 
each pixel of the back panel corresponds to a 2x2 neighborhood of the front panel. In the context of a multi-panel display, 
the term "front panel" or "top panel" refers to a panel nearer to the viewer's eye, and the term "back panel" or "bottom 
panel" refers to a panel farther away from the viewer's eye (i.e., behind one or more other panels). FIG. 10 shows a 
cross-section of a portion of a multi-panel display assembly 200, in the present example being a two-panel stack, in 
which each pixel of the first panel 202, in the present example being a back panel, corresponds to two pixels of the 
second panel 204, in the present example being the front panel, in the illustrated dimension. 

[0117] The display assembly 200 may be a stack of panels which may include more than two panels 202, 204. 
Implementationsofthecorrectionmethodmay be applied to any type of panel: in-planeswitching (IPS), vertical alignment, 
etc. One panel may have a higher contrast ratio than another. For example, a back panel may have a higher contrast 
ratio than a front panel. 

[0118] In a multi-panel display assembly 200 having panels of different resolutions, a defect may occur in a higher- 
resolution layer and/or in a lower-resolution layer. For a defect in a higher-resolution layer (e.g., a front layer), a corre- 
sponding compensation and/or masking in a lower-resolution layer should account for its effect on other pixels in the 
higher-resolution layer. For a defect in a lower-resolution layer (e.g., a bottom layer), modification of a drive signal to a 
corresponding multi-pixel area in a higher- resolution layer may be needed. For a multi-panel display, the superposition 
function vmay be expressed as 



v^ExPSF(x\y) + 



where L/ denotes a masking luminance value superposed on the masking pixel M/With relative location (Xj,yj). Such a 
masking luminance value may be created by varying the transmittances of the component pixels that affect the display 

output at pixel Mj. 

[0119] FIG. 12a illustrates a simplest case of a two-panel display in which each of the two aligned panels has the 
same resolution and each pixel is illuminated by a perfectly collimated backlight source. In this case, /./may be expressed 
as the product of the light intensity of the backlight source and the masking portions of the transmittances of the two 
component pixels, which depend on the respective driving levels. (The total transmittance of a component pixel may be 
expressed as a sum of an image portion, which corresponds to a value indicated by a display signal, and a masking 
portion, which may be calculated using expression (1), for example.) Although the luminance factor Ly may include 
contributions from multiple subfactors, only the PSF of the top panel is considered, as this panel is the one where the 
image to be viewed is formed. 

[0120] As shown in FIG. 12b for embodiments using a backlight source, in practice the backlight source is often not 
perfectly collimated, such that light transmitted by a pixel on a back panel affects more than one pixel in a panel above 
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it. In this case, /_/ may be expressed as the product of (A) the masl<ing portion of the transnnittance of top pixel M^and 
(B) the sunn of the products of the masking portions of the transmittances of the underlying pixels affecting pixel M/ with 
the respective light intensities. For a case in which a defect in a lower layer causes multiple pixel errors in the top panel, 
superposition function v may be expressed as 



corresponding to a multiple-defect case as discussed above in relation to expression (3). Again, the PSFs in this ex- 
pression correspond to PSFs of pixels of the top panel. 

[0121] A multi-panel display may also be configured to have both color and monochrome panels. For example, such 
a display may be configured to have a color panel in front and a monochrome panel in back. Monochrome panels are 
typically less expensive (e.g., because no color filters are needed). The lack of color filters for at least one panel may 
also support a brighter display for the same backlight output. In such case, it may be desirable to modify the defect 
compensation and/or masking method to account for the effect that a defect in, and/or modulation of, the monochrome 
layer may affect the perception of all colors of the corresponding area of a color layer. 

[0122] A multi-panel display may also include one or more optical components between panels. Such an optical 
component may affect the impact of modulation of a pixel behind the component (e.g., by affecting the spatial area 
affected by the modulation). For example, a diffuser disposed between two panels may help to avoid moire effects arising 
from interference between the pixel structures of the panels. Such a diffuser may also cause modulation of one or more 
pixels in a back panel to affect a larger corresponding area in a front panel. It may be desirable for a defect compensation 
and/or masking operation to account for the effect of such optical components. 

[0123] In a single-panel display, maskabiiity of some defects maybe limited by the driving levels of the masking pixels. 
For example, applying a calculated masking luminance value may in some cases require driving a pixel brighter than 
saturation or darker than an off state. The use of multiple panels reduces the probability of saturation of any of the panels, 
as it is unlikely that, for any one area of the screen display plane, corresponding areas of two or more panels will be 
driven close to saturation. For example, it is uncommon in image display that an entire area of the screen is extremely 
bright, as this could create an almost blinding effect. Therefore, a back panel carrying lower-frequency information is 
unlikely to be driven close to saturation, even if the higher-frequency portion of the image includes some extremely bright 
areas. Similarly, it is unlikely that, for any one area of the screen display plane, corresponding areas of two or more 
panels will be driven at a near-zero level. A multi-panel arrangement may thus allow opportunities for compensation 
and/or masking of some defects that cannot be sufficiently compensated and/or masked in a single-panel display as- 
sembly. 

[0124] Another possibility is to reserve a margin on the drive level of one or more of the stacked panels so that the 
nonnal driving levels of the masking pixels do not impose as much of a limit on the possibilities of compensation anymore. 
For example, if during normal operation the back panel would only be driven up to maximum 90% drive level (that is to 
say, 1 00% drive level of the front panel and 90% drive level of the back panel corresponds to maximum white of the 
panel combination), then the remaining 10% can be used to drive the masking pixels as indicated by the compensation 
algorithm, which should be sufficient to be able to drive the masking pixels as needed by the compensation algorithm 
in nearly all cases. The same principle is valid for compensating white defects on almost black backgrounds: if one takes 
a margin at the lower video levels on one or more of the panels, then this margin can be used to make sure that masking 
pixels can always be driven sufficiently low even if the surrounding area is black. In a further example, margins are 
reserved at both the high and low ends of the driving level range (e.g., 5% at each end). When using stacked panels, 
the advantage is that while taking such a margin lowers the contrast, the remaining contrast still is sufficient and much 
higher as compared to a single-panel system. In single-panel systems, taking such a margin would typically result in 
panels having insufficient contrast and/or brightness. 

[0125] Each panel of a multi-panel display has a characteristic curve relating drive voltage to light output. The light 
output from the display stack at any particular time during operation may be expressed as the product of the light outputs 
(or transmittances) of the individual panels for the current driving voltages of the panels (with some allowance for other 
factors, such as loss due to intervening layers). 

[0126] A defect compensation and/or masking method in a multi-panel display may be adapted to take advantage of 
the principle that a particular light output result may usually be obtained by any of several (typically many) different 
combinations of driving levels for the various panels. For example, it may be possible to compensate for a defect in one 
panel by altering the drive of one or more corresponding pixels in another panel. 

[0127] In one such example, a top pixel that is ten grey levels too dark is compensated by brightening an underlying 
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back pixel. For a case in which the back pixel corresponds to more than one top pixel, the effect of such brightening on 
other top pixels may be connpensated by darkening them appropriately. In such manner, a desired image result may be 
obtained without performing any masking. Any combination of such compensation with masking may also be used, as 
in some cases the defect may be too severe for the available level of compensation to adequately correct it. 

5 [0128] The respective pixel drive levels used to obtain a particular display output level may also be adjusted to allow 
more latitude in defect compensation and/or masking. If It is desired to modulate pixels in an area of a panel that is close 
to saturation, for example, then another panel may be driven brighter in that area to allow the drive signal for the first 
panel to be reduced. The resulting increase in latitude may be sufficient to allow a set of masking values sufficient to 
mask the defect to be applied to masking pixels In the first panel. 

10 [0129] Another type of HDR display uses a spatially modulated backlight. Such a backlight may be implemented as 
an array of separately controllable point sources, such as an array of LEDs and/or an active-matrix organic LED (OLED) 
or polymeric LED (PLED) panel. One example of such a backlight includes several hundred small LEDs that are arranged 
in a plane parallel to and behind an LCD panel and are modulated individually and/or in groups according to the local 
content of the image being displayed. Some particular examples include LEDs that have a diameter of 5 millimeters, 

15 LEDs that have a diameter of 12 millimeters, LEDs that are packed in square or other quadrilateral (e.g., diamond) 
configuration as shown In FIG. 11 a, and LEDs that are packed In a hexagonal configuration as shown in FIG. 11 b. 
Each point source of the backlight and each pixel or subplxel of the panel may be configured to operate at any of a 
discrete number of luminance levels (e.g., 256 or 1024), which may be linear or nonlinear (e.g., with increasing step 
size as luminance Increases). The backlight and panel may differ In the number and/or linearity of the operating luminance 

20 levels. In some cases, the backlight may be configured to flicker In synchronism with the panel (e.g., to reduce motion 
blur), which flickering may occur across the entire display or within only a currently selected portion of the display. 
[0130] Another possible implementation of a spatially modulated backlight includes a plurality of (e.g., eight or twelve) 
fluorescent lamps and/or tubes (such as cold-cathode or hot-cathode fluorescent lamps (CCFLs, HCFLs)) that can be 
individually modulated. Such a fluorescent backlight allows spatial modulation of the luminance and/or color in one 

25 direction (1 D). It may be desirable to use lamps whose light outputs may be modulated rapidly enough to avoid visible 
flicker (e.g., at a modulation rate at least equal to a frame rate of the panel). FIG. 13a and FIG. 13b show examples of 
such a backlight 250. Other configurations of a fluorescent backlight may use U-shaped tubes instead. 
[0131] The spatially modulated backlight may be monochrome. For example, such a backlight may be made of white 
LEDs. Alternatively, the spatially modulated backlight may Include point sources of different colors. It Is also possible 

30 for such a backlight to Include one or more point sources (e.g., an array of point sources) each having two or more 
different colors. A source having different colors may be implemented as a die having multiple color emitters. In some 
cases, it may be possible to approximate such a source as a point source. For example, the source may include an 
optical system such as a microlens. Alternatively, such a source may Include a group of two of more packages such 
that It may be desirable for a defect compensation and/or masking method to account for the different spatial locations 

35 of the various colors. Some typical examples of the different colors within a multicolor source Include red, green, and 
blue (RGB); red, green, blue, and green (RGBG); red, green, blue, and white (RGBW); and red, blue, blue-green, and 
yellow-green. 

[0132] One or more of the point sources of a spatially modulated backlight may be defective (e.g., may be always on 
or always off, or may otherwise have a response to its driving signal that is visibly different than that of other point sources 

40 in the backlight of the same color or type). In a backlight having several hundred LEDs, for example, it is not uncommon 
for at least one of the LEDs to be defective. A defect compensation and/or masking method may be applied to an HDR 
display having a spatially modulated backlight in a comparable manner to that of a multi-panel display having a low- 
resolution back panel as described above. A defect compensation and/or masking method as described herein may 
also be applied to mask failure of a light source within a spatially modulated backlight. Such a failure may be modeled 

45 as a defective pixel, such that panel pixels are modulated to mask the defective light source. 

[0133] A failure of an individual LED in a spatially modulated backlight may affect the luminances of many pixels of 
the panel or panels above that defect. In other words, the light of a single LED will typically serve as backlight for multiple 
pixels of the panel or panels. More generally, the radiation pattern of a single LED of the backlight is typically such that 
it will influence the luminance and/or color point of more than one pixel of the panel or panels that are combined with 

50 the backlight. FIGURE A3c shows an example of backlight sources having overlapping radiation patterns and pixels 
that transmit light from multiple backlight sources. In such case, a masking luminance value L/may depend on several 
or many underlying backlight source drive levels. In the limit, the light intensity for a pixel may be a sum of contributions 
of all of the Individual sources in the backlight. Practically, the light intensity for a pixel may be limited to a sum of 
contributions of the two, three, or four closest backlight sources, such as an average of these sources. 

55 [0134] A defect compensation and/or masking method may also be adapted for use with various types of three- 
dimensional (3-D) displays. Potential applications for3-D displays include viewing synthetic and/or reconstructed volumes 
in three dimensions, which may be useful in such diverse fields as medical imaging (e.g., mammography, tomosynthesis, 
surgery visualization, etc.), aircraft collision avoidance, and engineering design. 
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[0135] Stereoscopic displays require the viewer to wear some type of apparatus to perceive tine 3-D image. Sucli a 
display may be implemented as a time-multiplexed two-dimensional display, where the viewer wears special glasses 
having shutters synchronized to the display. In this case, a defect compensation and/or masking algorithm as described 
herein may be applied to a defect in one of the shutters. Alternatively, such a display may be implemented using panel 
illumination that is time-multiplexed between two orthogonal polarizations, where the viewer wears glasses having 
different polarizers. In this case, a defect compensation and/or masking algorithm as described herein may be applied 
to a polarization-dependent defect in the panel. In another alternative, a head-mounted display includes a different 
display panel (e.g., a small LCD panel) for each eye. In this case, a defect compensation and/or masking algorithm as 
described herein may be applied to a defect in a panel. 

[0136] Autostereoscopic displays 300 provide an image having a three-dimensional appearance without the need for 
special glasses or other headgear. Such a display 300 includes a display panel 302 or spatial light modulator (e.g., an 
LCD panel) and a view splitter 304 disposed between the panel or modulator and the viewer. For example, the view 
splitter 304 may be implemented as a splitting screen disposed parallel to and at some distance (typically a few millimeters) 
in front of the display plane of the panel 302. The splitting screen is configured such that some pixels of the panel can 
only be viewed by one eye, and other pixels of the panel can only be viewed by the other eye. 

[01 37] It may be desirable for light entering the view splitter 304 to be collimated. I n one example, the panel is illuminated 
using laser light. In a more common example, an autostereoscopic display 300 includes a collimator disposed between 
the panel 302 and view splitter 304, or between the panel and the backlight. Such a collimator may be implemented as 
a Fresnel lens or a filter configured to transmit light only with a certain narrow range of angles in at least one dimension, 
such as a plate with holes or slits or a multilayer interference filter. 

[0138] One type of splitting screen 304 (a blocking screen) may be implemented to include a parallax barrier. For each 
eye, the barrier blocks light from certain pixels from reaching that eye. A parallax barrier is implemented to have a pattern 
of parts that pass light (i.e., reflective or transmissive regions) alternating with parts that block light (i.e., absorptive or 
opaque regions). Typically the widths of the blocking and passing parts are on the order of the width of a pixel of the 
LCD panel. 

[01 39] A fixed parallax barrier may be implemented as an optical sheet with a grating, or as a patterned mask having 
an alternating series of black vertical lines and clear slits or having some other pattern. FIG. 14a shows one example 
of such a fixed parallax barrier 304. Alternatively, a dynamic parallax barrier 314 may be implemented as a spatial light 
modulator (such as another LCD panel) in which a blocking pattern is formed by brightening and darkening pixels to 
form the passing and blocking parts and in which the blocking pattern may be changed. FIG. 14b shows one example 
of such a dynamic parallax barrier. 

[01 40] Another type of view splitter 304 is configured to divert light from portions of the display 302 to one or the other 
of the viewer's eyes. One type of a diverting splitting screen is a lenticular screen 324. Such a screen includes an array 
of microlenses that directs the light from each pixel to one eye or the other. The lens structure may be fixed, such as an 
array of cylindrical lenses (e.g., a one-dimensional array of lenses that extend vertically from top to bottom of the display 
and have semi-cylindrical cross-sections). FIG. 15 shows one example of a panel having such a fixed lens structure, 
and FIG. 16a shows an example of diversion of light emanating from adjacent pixels 350 to form two separate views. 
Anothertype of fixed lens structure is an array of round or integral lenses (e.g., a two-dimensional array of lenses having 
round cross-sections in a plane parallel to the display plane). 

[01 41 ] Alternatively, the lens structure may be dynamic, such as an array of controllable lenses. The array of controllable 
lens may be implemented to include liquid lenses. Such a lens may be configured to operate according to electrowetting 
(i.e., in a lens cell, varying the contact angle of an interface between two substances to the boundary of the cell according 
to an electrostatic potential). In one type of liquid lens, the index of refraction is varied by varying a voltage to change 
the shape of a meniscus. Alternatively or additionally, the array of controllable lenses may be implemented to include a 
material whose index of refraction varies according to an applied voltage. For example, the refractive index of a layer 
of liquid crystal material changes with orientation of the crystals. In one implementation of an array of controllable lenses, 
a liquid crystal material is embedded in a glass structure, and a voltage is selectively applied to locally alter the material's 
index of refraction. The array of controllable microlenses may include a passive birefringent microlens component index- 
matched with an adjacent layer (such as an isotropic polymer) for one polarization (also called a "polarization activated 
microlens"), such that one polarization experiences a lensing effect but an orthogonal polarization does not, combined 
with a separate polarization switch (e.g., a liquid crystal panel) to determine whether or not the viewer views the display 
(or a portion thereof) according to the lensing effect. 

[0142] A lenticular screen 324 may be configured to support images having horizontal parallax only. For example, a 
one-dimensional array of vertical cylindrical lenses provides horizontal parallax only. Alternatively, a lenticular screen 
may be configured to support images having both horizontal and vertical parallax. For example, a two-dimensional array 
of round lenses may support both horizontal and vertical parallax if an appropriately coded image is displayed on the 
underlying panel. 

[01 43] A dynamic lenticular screen may be configured to be switchable between a two-dimensional mode and a three- 
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dimensional nnode. In some cases, the screen is configured to be locally switchable (e.g., from one lens to the next) 
between 2-D and 3-D modes. In one application of such a screen, a 2-D desktop display having control elements (such 
as a taskbar, GUI, and/or buttons, etc.) also Includes a resizable and/or draggable window that presents a 3-D display. 
Typically the resolution and/or brightness of such a display will be higher In areas operating In the 2-D mode than In 
5 areas operating in the 3-D mode. Such a screen may be controlled to switch the display mode locally to two-dimensional 
at the location of a defect. 

[0144] An autostereoscopic display may be configured such that the set of pixels visible to one of the viewer's eyes 
is separate from the set of pixels visible by the viewer's other eye. Therefore a defect compensation and/or masking 
method as described herein may be adapted for use with an autostereoscopic display by performing a different minimi- 
se zation calculation for each eye, with each calculation being performed over a different corresponding set of masking 
pixels. Pixels that are adjacent within each set of masking pixels (i.e., pixels that are perceived by the viewer's eye to 
be adjacent) may not actually be adjacent on the display panel, such that the neighborhood over which the minimization 
calculation is performed maybe discontinuous in one or both dimensions of the display plane of the panel. Mathematically, 
therefore, a display pixel in a two-view autostereoscopic display can be described by means of two PSFs (point spread 
f5 functions), one for the left eye and one for the right eye. In the case of a multiview display, a pixel can be assigned as 
many PSFs as there are views available. Expression (1) may be extended to such a case as follows: 
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Where Vj=Ex PSFj^ ix',y') + 



"Zc^^PSF^ix'-x^y'-yD 



fori <j<n 



(for a two-view display, n = 2), PSFj^ Indicates the PSF for the defect for view y, and PSF^y Indicates the PSF for the /-th 
masking pixel for view y. Examples of f(v^, V2,...,v^ x',y') include (Svy)^, IZvyl, Zv^j/^l(x)^+(y))^ and (Ev^)2V(x')2 + (y)2 for 
1 <y< n, although the cost function f Is not limited to these examples. If there Is no crosstalk In the stereoscopic display, 

30 then one of the PSFs would be all zero (no light falling on that eye) while the other PSF (or PSFs) will actually describe 
the light of that pixel falling on the other eye (or the other views). In case there is crosstalk between the views, then the 
PSFs will be non-zero in the views that suffer from crosstalk. As a practical matter, crosstalk usually exists In autoster- 
eoscopic displays. In such cases, the cost function f may be configured to use one or more views to partially compensate 
for an error or errors In one or more other views. In another example, an error In only one view of a stereo pair may be 

35 very distracting to the viewer, and In such a case the cost function f may be configured to introduce a related error In 
the other view of the stereo pair. 

[0145] As described above, a defect compensation and/or masking method may be used to calculate correction 
luminance values for a set of masking pixels. Additionally or alternatively, a defect correction method as applied to an 
autostereoscopic display may include altering the splitting screen to mask a defect. In the case of a parallax barrier, 

40 such a method may include altering the barrier to allow one view to receive light from a pixel of the other view. Such a 
result may be achieved, for example, by removing at least some of the barrier part that blocks light at that location. For 
such a case in which an overlap exists between the sets of pixels visible to each eye. It may be desirable for the two 
minimization calculations to observe one or more mutual constraints relating to the commonly visible pixels. 
[0146] A luminance defect in a 3-D display may also create an inconsistency in the binocular image that can be very 

45 distracting to a viewer. If the view for one eye contains a defect that Is not present In the view for the other eye, such 
that one eye sees the intended luminance but the other eye sees an abnormal luminance, a disturbing depth cue error 
may result. Such a view depicts a situation that is not physically possible, and the defect may have an effect on the 
viewer that is disproportionate to the affected area. Such distractions may be especially undesirable for applications 
such as imaging for medical diagnosis. Depth cue errors may arise in autostereoscopic displays as well as in some 

50 stereoscopic displays such as head-mounted displays (e.g., from a defect In one of the displays) and displays that use 
alternating polarizations (e.g., from a polarization-dependent defect in a pixel and/or a defect In one of the shutters). 
[0147] In some cases of depth cue error, the defect may be sufficiently masked by applying a defect compensation 
and/or masking method as described herein. In other cases, some depth cue error may remain due to a defect that 
cannot be fully compensated and/or masked. In these cases, it may be desirable to artificially introduce one or more 

55 defects to mask the remaining depth error. For example, an artificial luminance and/or color defect may be Introduced 
in one view to the extent that a corresponding luminance and/or color defect In the other view remains uncorrected. In 
one example, a defect of sixty grey levels can only be corrected to forty grey levels, such that a defect of twenty grey 
levels remains. In this case, an artificial defect of twenty grey levels Is Introduced Into a corresponding location of the 



19 



EP 1 923 860 A2 



other view to compensate. A defect created in such nnanner that is consistent across the two views nnay be perceived 
as dust or sonrie other nninor imperfection and should be less disturbing to the viewer than a defect that creates a scene 
which is physically impossible. 

[0148] A defect compensation and/or masking method may be configured to introduce an artificial defect in one view 
5 to the extent of an uncorrected defect in the other view as described above. A defect compensation and/or masking 
method may also be configured to perform a calculation that minimizes a depth cue distraction penalty, in addition to or 
in the alternative to minimizing a luminance and/or color defect penalty as described above. 

[0149] The 3-D image of an autostereoscopic display is typically visible only within a relatively small "sweet spot," 
which is normally only about five to ten centimeters wide at a specified viewing distance from the panel (e.g., 70 cen- 

10 timeters). To some extent, this sweet spot may be moved laterally (and/or vertically, if the display supports vertical 
parallax) by changing the image displayed on the panel. It may be desirable for an installation including an autostereo- 
scopic display as described herein to also include a head-tracking apparatus such that the displayed image may be 
configured to move the sweet spot according to the current location of the viewer's eyes. For example, an autostereoscopic 
display may be configured to dynamically control the splitting screen to redirect the different views according to the 

15 predicted or detected current eye position. 

[0150] Examples of head-tracking apparatus include ultrasonic ranging devices that locate the viewer's head. Other 
examples include visible-light and/or infrared imagers that track the position of the viewer's head and/or eyes. Such 
apparatus may be mounted to the display or independently mounted. 

[0151] One of the more effective types of head-tracking apparatus is a device in which two images of the viewer's 
20 face are taken at the same time or within a short time period. One of the images is taken under normal illumination and/or 
by a camera sensitive only to visible light. The other image is taken under infrared illumination and/or by a camera 
sensitive to infrared light. The two images are compared (e.g., subtracted) to determine the location of the viewer's eyes, 
as the eye tends to reflect infrared light much more than other parts of the body. One form of such a device is called a 
"blinking infrared" head-tracker because it flashes infrared illumination for alternate images. 
25 [0152] An autostereoscopic display may also be implemented as a multiview display that simultaneously generates 
a 3-D view for more than one sweet spot at a time. Resolution of such a display usually decreases in proportion to the 
number of different views. FIG. 1 6b shows one example of a generation of four different views from adjacent pixels 350. 
Alternatively, a multiview display implemented with a dynamic splitting screen may be configured to operate the panel 
and screen in synchronism for time-multiplexed generation of multiple views. A multiview display may also be combined 
30 with head-tracking (e.g., individual head-tracking of multiple viewers). A defect compensation and/or masking method 
as described herein may be adapted for application to a multiview display (for example, by performing a different pair 
of minimization calculations for each view). 

[0153] The above-described method embodiments of the present invention may be implemented in a processing 
system 500 such as shown in Fig. 17. Fig. 17 shows one configuration of processing system 500 that includes at least 

35 one programmable processor 503 coupled to a memory subsystem 505 that includes at least one form of memory, e.g., 
RAM, ROM, and so forth. It is to be noted that the processor 503 or processors may be a general purpose, or a special 
purpose processor, and may be for inclusion in a device, e.g., a chip that has other components that perform other 
functions. Thus, one or more aspects of the present invention can be implemented in digital electronic circuitry, or in 
computer hardware, firmware, software, or in combinations of them. The processing system may include a storage 

40 subsystem 507 that has at least one disk drive and/or CD-ROM drive and/or DVD drive. In some implementations, a 
display system, a keyboard, and a pointing device may be included as part of a user interface subsystem 509 to provide 
for a user to manually input information. Ports for inputting and outputting data also may be included. More elements 
such as network connections, interfaces to various devices, and so forth, may be included, but are not illustrated in Fig. 
1 7. The various elements of the processing system 500 may be coupled in various ways, including via a bus subsystem 

45 513 shown in Fig. 1 7 for simplicity as a single bus, but will be understood to those in the art to include a system of at 
least one bus. The memory of the memory subsystem 505 may at some time hold part or all (in either case shown as 
511) of a set of instructions that when executed on the processing system 500 implement the steps of the method 
embodiments described herein. Thus, while a processing system 500 such as shown in Fig. 17 is prior art, a system 
that includes the instructions to implement aspects of the methods for displaying images in a display assembly is not 

50 prior art, and therefore Fig. 1 7 is not labelled as prior art. 

[01 54] The present invention also includes a computer program product which provides the functionality of any of the 
methods according to the present invention when executed on a computing device. Such computer program product 
can be tangibly embodied in a carrier medium carrying machine-readable code for execution by a programmable proc- 
essor. The present invention thus relates to a carrier medium carrying a computer program product that, when executed 

55 on computing means, provides instructions for executing any of the methods as described above. The term "carrier 
medium" refers to any medium that participates in providing instructions to a processor for execution. Such a medium 
may take many forms, including but not limited to, non-volatile media, and transmission media. Non volatile media 
includes, for example, optical or magnetic disks, such as a storage device which is part of mass storage. Common forms 
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of computer readable media include, a CD-ROM, a DVD, a flexible disk or floppy disk, a tape, a memory chip or cartridge 
or any other medium from which a computer can read. Various fonns of computer readable media may be involved in 
carrying one or more sequences of one or more instructions to a processor for execution. The computer program product 
can also be transmitted via a carrier wave in a network, such as a LAN, a WAN or the Internet. Transmission media can 
take the form of acoustic or lightwaves, such as those generated during radio wave and infrared data communications. 
Transmission media include coaxial cables, copper wire and fibre optics, including the wires that comprise a bus within 
a computer. In one aspect, the computer program product may be a controller for performing the method for reducing 
visual impact of defects for a multi-layer or multi-panel display assembly, separate from or incorporated in the display 
assembly. 

[0155] The foregoing presentation of the described embodiments is provided to enable any person skilled in the art 
to make or use the present invention. Various modifications to these embodiments are possible, and the generic principles 
presented herein may be applied to other embodiments as well. For example, an embodiment may be implemented in 
part or in whole as a hardwired circuit, as a circuit configuration fabricated into an application-specific integrated circuit, 
or as a firmware program loaded into non-volatile storage or a software program loaded from or into a data storage 
medium (e.g., volatile or non-volatile semiconductor memory, one or more magnetic or optical disks, etc.) as machine- 
readable code, such code being instructions executable by an array of logic elements such as a microprocessor or other 
digital signal processing unit. Methods of defect compensation and/or masking as described herein may also be applied 
to other pixel-addressable display technologies such as plasma display panels (PDPs). Thus, the present invention is 
not intended to be limited to the particular embodiments shown above but rather is to be accorded the widest scope 
consistent with the principles and novel features disclosed in any fashion herein. 



Claims 

1. A display assembly (200) comprising a first imaging layer and a second imaging layer, wherein a drive signal to a 
pixel (14) on one of the first and second Imaging layers is based on a defect in the other of the first and second 

imaging layers. 

2. A display assembly (200) according to claim 1 , wherein at least one of the first and second imaging layers is a 
spatially modulated backlight (250). 

3. A display assembly (200) according to any of claims 1 or 2, wherein said assembly (200) includes a third imaging 
layer disposed between said first and second imaging layers. 

4. A display assembly (200) according to claim 3, wherein each of said first and third imaging layers comprises a liquid 
crystal panel. 

5. A display assembly (200) according to any of the previous claims, the display assembly comprising a first panel 
(202) in the first imaging layer and a second panel (204) in the second imaging layer. 

6. A display assembly (200) according to claim 5, wherein at least one of the first and second panels (202, 204) is a 
liquid crystal display panel. 

7. A display assembly (200) according to any of claims 5 to 6, wherein the resolution of the second panel (204) differs 
from the resolution of the first panel (202). 

8. A display assembly (200) according to any of claims 5 to 7, wherein the display assembly (200) includes a backlight 
(250) and wherein the second panel (204) is disposed between the first panel (202) and the backlight (250). 

9. A method for displaying images in a display assembly (200) comprising at least a first and second imaging layer, 
the method comprising providing a drive signal to a pixel (14) on one of the first and second imaging layers based 
on a defect in the other of the first and second imaging layers. 

10. A method according to claim 9, wherein the method comprises modulating pixel data in a neighbourhood around a 
defect of a display panel in a stack of at least two panels (202, 204), wherein the neighborhood includes at least 
one pixel of another panel in the stack. 

11. A method according to claim 9, wherein the neighbourhood only includes pixels that lie along a single straight line 
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in space. 

12. A method according to any of claims 9 to 11 , the method comprising calculating a pixel drive signal according to a 
model of a response of a human visual system. 

13. A method according to claim 12, wherein the line is perpendicular to the display plane of the panel (202). 

14. An autostereoscopic display assembly (300) comprising a display panel (302) and a view splitter (304), wherein 
said view splitter (304) is configurable during operation according to a defect of the display panel (302). 

15. An autostereoscopic display assembly (300) according to claim 14, wherein said view splitter (304) includes a 
parallax barrier. 

16. An autostereoscopic display assembly (300) according to claim 15, wherein said parallax barrier comprises a liquid 
crystal panel (314). 

17. An autostereoscopic display assembly (300) according to claim 14, wherein said view splitter (304) includes an 
array of controllable lenses (324). 

18. An autostereoscopic display (300) according to claim 17, wherein at least one of said array of controllable lenses 
comprises a material whose index of refraction is electrically controllable. 

19. An autostereoscopic display (300) according to any of claims 14 to 18, wherein said display (300) includes a head- 
tracking device, and wherein said view splitter (304) is configurable during operation according to an output of said 
head-tracking device. 

20. An autostereoscopic display (300) according to any of claims 14 to 19, wherein a principal surface of said view 
splitter (304) is disposed parallel to a display plane of the display panel (302). 

21 . An autostereoscopic display (300) according to any of claims 1 4 to 20, wherein said view splitter is disposed in front 
of a viewing surface of the display panel. 

22. A controller for controlling a display assembly for displaying images according to a method as described in any of 
claims 9 to 13. 

23. A computer program product for, when executed on a computing means, performing a method for displaying images 
in a display assembly according to any of claims 9 to 13. 

24. A machine readable data storage device storing the computer program product of claim 23. 

25. Transmission of the computer program product of claim 23 over a local or wide area telecommunications network. 
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